Applying big data technologies in the financial sector – using sentiment analysis to identify correlations in the stock market

The aim of this article is to introduce a system that is capable of collecting and analyzing different types of financial data to support traders in their decision-making. Oracle’s Big Data platform Oracle Advanced Analytics was utilized, which extends the Oracle Database with Oracle R, thus providing the opportunity to run embedded R scripts on the database server to speed up data processing. The extract, transform and load (ETL) process was combined with a dictionary-based sentiment analysis module to examine cross-correlation and causality between numerical and textual financial data for a 10 week period. A notable correlation (0.42) was found between daily news sentiment scores and daily stock returns. By applying cross-correlation analysis and Granger causality testing, the results show that the news’ impact is incorporated into stock prices rapidly, having the highest correlation on the first day, while the returns’ impact on market sentiment is seen only after a few days.